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(57) An image retrieval method with improved per- 
formance in which a data image similar to a query image 
is retrieved by appropriately combining colour and tex- 
ture features. The method for retrieving a data image 
similar to a query image in an image database contain- 
ing a plurality of data images involves: calculating colour 
and texture distances between a query image and each 
data image in the image database; weighting the calcu- 
lated colour and texture distances with predetermined 
weighting factors; calculating a feature distance be- 
tween the query image and each data image by com- 
bining the weighted colourand texture distances by con- 
sidering human's visual perception attributes; and de- 
termining the data image similar to the query image us- 
ing the feature distance. According to the image retrieval 
method, data images similar to a query image can be 
retrieved based on the human visual perception mech- 
anism by combining both colourand texture features ex- 
tracted from image regions. In particular, the image re- 
gion based retrieval enables much accurate retrieval of 
more objects and many kinds of information from a sin- 
gle image. 
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Description 

[0001] The present invention relates to a method of searching a database of images, the method comprising calcu- 
lating a feature distance between a query image and an image in the database to obtain a measure of the relevance 

5 of said image in the database, said feature distance being calculated by combining colour and texture feature distances. 
[0002] In recent years.a sudden increase in multimedia data has stimulated research in the fields of data retrieval 
and databases for efficient multimedia data management. In particular, research in the field of image retrieval has 
focused on an alternative to existing text-based image retrieval methods and other advanced image retrieval methods. 
For text-based data retrieval, a great effort needs to be made to retrieve the desired data from a large-scale database, 

10 and it is unsuitable for partial-feature-based retrieval or quantitative data retrieval. To address this drawback, there 
have been approaches to retrieval using numeric feature descriptors which well represent the features of multimedia. 

data. , 
[0003] Colour and texture features, which are dominant characteristics in human visual perception, can be expressed 
by feature descriptors. According to a conventional image retrieval method, an imagedatabase in which data images 

is are indexed by a feature descriptor is built up. and desired images are retrieved from the image database based on 
the feature descriptor. An example of a conventional colour feature based image retrieval method includes extracting 
colour vectors, quantizing the colour vectors, getting a representative colour vector from the quantized colour vectors, 
and retrieving image data using the representative colour vector. A conventional texture feature based image retrieval 
method uses the co-occurrence matrix, the Gaussian Markov Random Field (GMRF) models, etc. A recent method of 

20 texture feature extraction uses a Gabor filter that enables a multi-channel approach suitable for the human perception 
mechanism of visual images. . 
[0004] In the conventional image retrieval methods, colour and texture based data retrievals are separately carried 
out and colour and texture features are extracted from the entire image. For this reason, the conventional methods 
cannot be applied for an image having a plurality of objects therein or expressed in multiple colours and textures. Either 

25 the colour or texture features are available for retrieving particular image data. Thus, there is a problem of incorrect 
image retrieval results. 

[0005] To solve these problems, another conventional image retrieval method retrieves a target image in the vector 
space formed for both colour and texture vector components. Here, the vector space is formed through simple sum- 
mations of the colour and texture vector components, thereby increasing dimensionality of the vector space. Thus, 
30 there is a need to redefine a distance function to the vector space with the increased dimensionality and to calculate 
a weighting factor for each component, which is relatively burdensome. The computations are highly complex and the 
results may be incorrect. 

[0006] In general terms, the present invention provides a method for retrieving a data image similar to a query image 
using a feature distance calculated by combining colour and texture distances by considering human's visual perception 
35 attributes. 

[0007] Preferably, a method according to the present invention is characterised in that the colour distance and the 
texture distance are weighted by respective functions of both the colour entropy of the query image and texture sample 
areas such that the relative contributions of the colour distance and the texture distance to said feature distance sub- 
stantially correspond to the relative importance of colour and texture in human perception .of the similarity between 
40 said images. 

[0008] In one embodiment, the present invention provides a method for retrieving a data image similar to a query 
image in an image database containing a plurality of data images, the method comprising: (a) calculating color and 
texture distances between a query image and each data image in the image database; (b) weighting the calculated 
color and texture distances with predetermined weighting factors; (c) calculating a feature d.stance between the query 

45 image and each data image by combining the weighted color and texture distances by considering human's visual 
perception attributes; and (d) determining the data image similar to the query image using the feature distance. Pref- 
erably before step (a), the image retrieval method further comprises: (pa-1) segmenting the query image and each 
data image into a plurality of regions using color features; and (pa-2) determining sample regions in the query image 
and each data image for extraction of texture features. 

50 Preferably step (a) comprises: (a1) generating colour vectors of the plurality of regions using the colour features and 
calculating colour distances; and (a2) generating texture vectors of the plurality of regions using the texture features 
and calculating texture distances. . . 

[0009] Preferably, step (pa-1 ) comprises quantizing colour vectors of the query image and each data image. This 
step of quantizing the colour vectors may comprise: (pa-1-1) performing peer group filtering on the query image and 

55 each data image for noise removal and smoothing effects; and (pa-1 -2) clustering filtered pixel values of the query 
image and each data image using the generalised Lloyd algorithm. 

[0010] Preferably, the image retrieval method further comprises: defining a J-value indicating colour uniformity in 
each pixel of the query image and each data image which have undergone quantization; storing the J-value in each 
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pixel of the query image and each data image to obtain J-images; segmenting each J-image into a plurality of regions 
by a predetermined segmentation method; repeating the segmentation of each J-image to obtain a map of over-seg- 
mented regions for each J-image; and obtaining a final map for each J-image by merging the over-segmented regions 
based on correlation of colour; 

5 [0011J Preferably, step (pa-1-2) comprises applying a predetermined algorithm to increase the number of resulting 
clusters or to merge the resulting clusters. 

[0012] In the image retrieval method, the colour features are expressed by a colour feature descriptor f c (l k ) with a 
representative colour vector and the percentage of the representative colour vector for each region. The colour feature 
descriptor f c {l k ) may be expressed by: 

10 

where k is a positive integer indicating the serial number of each region, c ki is the i-th representative colour vector of 
15 the k-th region (/ = 1,2,..,/V/) ( p*, is the percentage of the i-th colour representative colour vector in the k-th region, and 
N k is the number of the representative colour vectors in the k-th region. 

[0013] Preferably, the image retrieval method further comprises indexing a feature vector space by a representative 
colour and its percentage in each segmented region. This step of indexing the feature vector space may comprise: 
assigning representative colours to grid points in a colour space having a grid structure; and storing the result of the 
20 assignment as a table in a database. 

[0014] In the image retrieval method, the colour distance between each colour vector of each region of the query 
image and each colour vector of each region of each data image may be calculated using: 

25 

*ci*M = Z Pi + I Pi - X L 2a U,2j PuPlJ 



30 

where, supposing that T d is the maximum distance by which similarity of two colours is determined, a is a predetermined 
scaling coefficient, d max = a T d , and d fJ is the Euclidean distance ||c,. - cj| between two colour vectors c,- and c yV a, y = 1 - 
-J— if djj < r d . and a, y = 0 if d i} > T d . 

f(Wt5] In the image retrieval method, preferably, step (a2) uses the Gabor function. In step (a2). the texture vectors 
35 of the plurality of regions may be generated using the Gabor function having N frequency channels and M orientation 
channels, where N and M are predetermined positive integers. 

[0016] In the image retrieval method, the texture features are expressed by a texture feature descriptor f t (l k ): 

4Q f t U k ) M(™* ir O-/cii)'( m *12'°*12> (™MM'°*lMM m *2V°*21.) <W ^NM^RNm)) 

where m kij is the means value of the pixel values of the i-th frequency channel and j-th orientation channel for region 
l k , and o kij is the deviation of the pixel values of the i-th frequency channel and the j-th orientation channel for region 
l k . In step (a2), the texture distance between each colour vector of each region of the query image and each colour 
45 vector of each region of each data image may be calculated using: 



«*,(/.. A)- Z I 

50 i=l j=) 



[0017] Preferably, step (pa-2) comprises: (pb-1 ) obtaining a grid map of the query image and each data image; and 
(pb-2) obtaining a texture sample of a desired size for each region based on the grid map. In this case, step (pb-1) 
55 may comprise: (pb-1-1) taking a rectangle M(iJ) having largest side lengths for a sample region, where 0 # I # n and 
0 # j # m; (pb-1-2) dividing the rectangle M(iJ) into sub-rectangles each having a / x / size; and (pb-1 -3) obtaining a 
grid map for the sample region using a predetermined function which outputs 1 if all the divided grids belong to the 
sample region, and otherwise, outputs 0, the predetermined function expressed by: 



♦is 



3 



EP 1 221 662 A2 



if P g {x,y)cI k forV(x 9 y) 
otherwise 



where 0</< 

n 

.7 

-1 , 0<j< ■ ' 

m 

.7 

-1, 0<x</-1 , 0<y</- 1, and p,/x,y) is a point that belongs to rectangle M{H + x, jl + y). Preferably, after step (pb-1-3), the 
image retrieval method further comprises: (pb-1 -4) transforming the grid map obtained in step (pb-1-3) to a grid distance 
map by repeating computations with: 

{^ritf('./)} 0 = Mgridi'j)' 

and 

{M d grid (iJ)} n = min({ <,,,(/ j-1)}^ . {<,*(M J)}^ .{< rfd (/7+1 .{<w(/+1 J)}^)^ ■ 

and 

(pb-1-5) performing region growing in every direction from a seed of the grid map, which has the largest distance in 
the grid and is expressed by 

(a,b) = arg max {h/f* (/V)}, to satisfy the relation h/f* . (/'.;') > 0 until the maximum area rectangle is extracted. 

[0018] fc/eferabfy the image retrieval method fffitner comprise fitting the maximum area rectangle extracted in step 

(pb-1-5) to the desired size of the texture sample by wrapping. Alternatively, the image retrieval method may further 

comprise fitting the maximum area rectangle extracted in step (pb-1-5) to the desired size of the texture sample by 

mirroring. 

[0019] In the image retrieval method, step (b) may comprise: (b-1) placing the colour and texture distances in 2-di- 
mentional vector spaces defined by the respective colour and texture distances and their predetermined weighting 
factors; and (b-2) projecting the result of the projection onto the 2-dimensional vector spaces onto a 1-dimensional 
distance space based on the human visual perception mechanism. 

[0020] In the image retrieval method, preferably, the predetermined weighting factor to the colour distance is deter- 
mined based on the distribution of representative colours. More preferably, the predetermined weighting factor to the 
colour distance is determined by: 

N 1 

where p ■ is the percentage of the i-th representative colour of the query image region. 
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[0021] Preferably, the texture distance is defined by: 

d t V q J,) = a ' 1 d t (l q ,l,) 

where i q denotes the query image or each region of the query image, s denotes the sample region, A{) denotes the 
area of the sample region of a desired size, count( ) is the number of wrappings done to obtain the desired size of 
sample region, and a is a constant. 

[0022] Preferably, the predetermined weighting factor to the texture distance is determined based on the area of an 
initial sample region extracted from the query image and the area of an sample region extracted from each data image. 
More preferably, the predetermined weighting factor to the texture distance is determined by: 



where l q denotes the query image or each region of the query image, s denotes the sample region, A( ) denotes the 
area of the sample region of a desired size, count{ ) is the number of wrappings done to obtain the desired size of 
sample region, and a is a constant. 

[0023] Preferably, the image retrieval method further comprises, before step (b-1), normalising each of the colour 
and texture distances. In this case, normalising each of the colour and texture distances may be performed using 
Gaussian normalisation. Preferably, normalising each of the colour and texture distances comprises: under the as- 
sqmptioji that that sequence is a Gaussian sequence/performing normalisation within the range of [!1 , 1] using v m k 
_ m.k ■ k baseci on tne mean s value u^ and deviation o k of sequence v^; and mapping the result of the normalisation 
into ?f7* range of [0, 1] using v = ^ m k+ . 

[0024] Preferably, normalising each of the colour and texture distance comprises normalising the texture distance 
by updating the u me a ans value u f and deviation o t of texture distances by excluding a largest texture distance until the 
condition of /<x— <-^, where k is a constant, is satisfied. 

[0025] In the^macje retrieval method, step (b-2) may comprise projecting the colour and texture distances onto a 
1 -dimensional distance space using: 



co c (Of 

where W c = — , and W t = — 

[0026] An f e>nb c oyiment of trite pfe*sent invention will now be described, by way of example, with reference to the 
accompanying drawings, in which: 

Figure 1 is a flowchart illustrating the major steps of a preferred embodiment of an image retrieval method according 
to the present invention:. 

Figure 2 is a flowchart illustrating an example of indexing data images of an image database by colour feature; 

Figure 3A shows an example of an image segmented according to its colour features: 

Figure 3B shows an image expressed with quantized colour vectors for the image of Figure 3A; 

Figure 3C shows a J-image; 

Figure 3D shows an image after segmentation is completed; 

Figure 4 is a flowchart illustrating indexing data images of an image database by texture feature; 
Figure 5 shows an example of the spatial frequency response equal to half of the 2-D Gabor function; 
Figure 6 is a flowchart of an example of taking a texture sample from an image region; 
Figure 7 illustrates a feature of the present invention; 

Figure 8A shows examples of data image regions for a given query image region, which are retrieved by combining 
colour and texture features; 
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Figure 8B shows examples of data image regions for a given query image region, which are retrieved based on 
only colour feature; and 

Figure 8C shows examples of data image regions for a given query image region, which are retrieved by considering 
texture feature on the candidate data image regions having significant colour distances. 

[0027] Referring to Figure 1 , the image retrieval method according to the present invention involves calculating colour 
and texture distances between a query image and each of the data images stored in the image database (Step 102). 
[0028] In the present embodiment, it is assumed that each of the data-images stored in the image database is seg- 
mented into a plurality of regions according to colour and/or texture features, and an image region that a user wants 
to search for in the image database exists in one data image segmented into a plurality of image regions. That image 
region which is searched for is referred to as a query image region. In the image retrieval method according to the 
preferred embodiment of the present invention, a data image similar to the query image region is retrieved. 
[0029] First, a step of extracting colour (feature) vectors for the query image and data images and calculating colour 
distances will be described. 

[0030] Referring to Figure 2, colour vectors for the query image and data images are quantized (Step 202). Figure 
3A shows an example of an image segmented according to its colour features, and Figure 3B shows an image ex- 
pressed with the quantized colour vectors for the image of Figure 3A. Next, peer group filtering is performed on the 
images with the quantized colour vectors for noise removal and smoothing effects (Step 204). Peer group filtering is 
described in WO-A-0046749, Next, filtered pixel values are clustered using the generalised Lloyd algorithm (GLA) 
(Step 206). Although the number of clusters is fixed for the GLA, a predetermined algorithm can be applied to increase 
the number of clusters or merge the clusters. 

[0031] Next, a J-value as an index of colour uniformity in each pixel after the quantization is defined (Step 208). For 
the images expressed by the quantized colour values, the J-value indicating pixel colour uniformity is stored in each 
of the pixels to obtain a J-image (Step 210). Figure 3C shows the J-image obtained in Step 210. The J-image is 
segmented into a plurality of regions by a predetermined method. The J-value based image segmentation is disclosed 
in WO-A-0065839. A map of over-segmented regions is obtained by iterating segmentation (Step 214). Next, a final 
map is obtained by merging segmented regions based on correlation of colour (Step 216), thereby completing the 
image segmentation. Figure 3D shows an image after the segmentation is completed. 

[0032] Next, a feature vector space is indexed by the representative colour and its percentage in each segmented 
region (Step 21 8). For example, a colour feature describer f c (l k ) of each segmented region is expressed using a rep- 
resentative colour vector and the percentage of the representative colour vector: 

f C Vk) = {(C*1.P W ).(C* 2 .P* 2 ) (C k N k ,P kNk )} ( 1 ) 

where k is a positive integer indicating the serial number of each region, c ki is the l-th representative colour vector (/ 
= 1,2,.., N) of the k-th region, p ki is the percentage of the i-th representative colour vector, and N k is the number of the 
representative colour vectors in the k-th region. Then, the representative colours are assigned to grid points in a colour 
space having a grid structure. Next, the indexed results are stored as a table in a database, thereby completing indexing 
of the feature vector space by the colour vectors. 

[0033] The distance between colour vectors of a query image region and a data image region is expressed by: 



N t * 2 *t N 2 

= Z Pi + Z Pi - Z Z 2 *tw PuPzj (2) 

/«l '=) i^l j=\ 



[0034] In formula (2), supposing that T d is the maximum distance by which similarity of two colours is determined, 
a is a predetermined scajjng coefficient, d max = a T d . and d fJ is the Euclidean distance \\c f - c ; || between two colour 
vectors c i and c jt a fJ = 1 2— if d {j < T d , and a,y = 0 if d fJ > T d . 

[0035] Next, a step of esmfccting texture (feature) vectors and calculating texture distances will be described. 
[0036] The reason for the wide use of multi-channel filtering in recent years as a texture analysis approach is that 
the similarity of this multi-channel .filtering to human visual perception is evidenced by psychophysical and psycho- 
physiological experiments. The mammalian visual tissue has a perception mechanism of tuning visual signals into 
narrow band frequencies and orientations. In other words, the visual perception mechanism of humans can be explained 
with a plurality of channels each having a bandpass filter with particular frequency and orientation angle domains. The 



EP 1 221 662 A2 

important factor in multi-channel filtering texture analysis is to define the functional characteristics of the channels and 
the number of channels. In addition, extraction of correct texture features from the image passed through the filters, 
the relation between each channel, and appropriate combination of texture features from different channels are also 
important considerations. The most effective function for modelling texture features is the Gabor function. 
[0037] Referring to Figure 4, images are filtered by Gabor filtering (Step 402). A 2-dimensional (2-D) Gabor function 
used for the Gabor filtering of images can be defined as a complex sinusoidal function, which is a modulation of a 2-D 
Gaussian function the spectrum of which is shifted in the frequency domain. This function is expressed by: 

h(x,y) = g f (x',/)exp[2^(t; x + V y )} (3) 
(x,y) = (xcos<|> + ysin<i> - xsint|> + ycos4>) (4) 



15 



1 



5 exp[-(x 2 /k 2 + y 2 )/2S 2 ] 



(5) 



20 



H(jj.v) = exp{-2ir 2 5 2 [(u f V) 2 



(/- V 1 ) 2 )} 



(6) 



25 



30 



35 



40 



where a. is the aspect ratio, and 5 2 is variance. 

[0038] Supposing that the radial bandwidth is B and the orientation bandwidth is I, the radial bandwidth B and the 
orientation bandwidth I of the Gabor filter are determined such that about half of the spatial-frequency response for 
each channel filter is the same to reduce redundancy in the Gabor filter. 

[0039] Referring to Figure 5, a first channel 502 has the same scale as a second channel 504, but different orientation. 
The first channel 502 has a different scale from a third channel 506, but the same orientation. Gabor filtering for an 
image is implemented by convolving the image with the Gabor function, thereby outputting different images through 
filters having different orientations and scales. 

[0040] The mean value and variance of the pixel values of the filtered image are calculated (Step 404). The mean 
value and variance of the pixel values obtained by convolution of the image with the Gabor filter represent texture 
features of the image. Next, the feature vector space is indexed by the mean value and variance of the pixel values 
(Step 406). The image segmented into a plurality of regions is expressed using the Gabor filter having N frequency 
channels and M orientation channels: 



Wk) = {( m *i1'°*1l)'( m *12' a *12) ( ,r, f(iM ,0 MM>'(^2r 0 fc2l)' 

(m kj j,O ki j) ( m RNM'°kNM) 



(7) 



45 



where m kjj is the mean value of the pixel values of the i-th frequency channel and j-th orientation channel for image 
region l k . and o MJ is the deviation of the pixel values of the i-th frequency channel and the j-th orientation channel for 
image region i k . 

[0041] A distance function for two image regions based on their texture vectors is expressed as: 



50 



a(m.-) 



N M 



cr(cr iy ) 



(8) 



[0042] According to the present invention, texture features are extracted from an image region having an arbitrary 
shape. An image region selected by a user has an irregular shape. Colour information is irrelevant to the shape of the 
image region because it is attainable from the number and distribution of colour pixels in the image region. Supposing 
that texture is defined as an irradiance pattern that can be discriminated by its proportion in the spatial frequency 
domain, the texture feature is affected by pixel patterns and the shape of image regions. It is not easy to apply the 
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Gabor filter that outputs the convolution of an image with the Gabor function to an image hav.ng an arbitrary shape. 
When an image is retrieved through the Gabor filter using texture samples having different sizes, matching efficiency 
is poor compared with an image retrieval based on the same-sized samples. For this reason, in the present embodiment, 
there is a need to take rectangular samples from image regions having an arbitrary shape, which are the same in size 
and represent the texture feature of corresponding regions. In this sampling process, the rectangular samples should 
be large enough to include the irradiance pattern of the pixels, such that accurate texture information is extracted from 

the relevant image regions. , 0 »«» cm\ ir, 

[0043] Referring to Figure 6, a grid map for the sample region is obtained to take a texture sample (Step 602). n 
particular, a rectangle M(i,J) (where, 0 < / < n and 0 <j < m), which has largest side lengths, is taken from the sample 
reqion Next the rectangle M(i.j) is divided into sub-rectangles each having a / x / size. Then, a gr.d map for the sample 
region'is made with the following predetermined function, which outputs 1 if all the divided grids belong to the sample 
region, and otherwise outputs 0: 

f 1 if P i} (*, y) c I k for V(x, y) 
M *"W\o otherwise "™ 



where 0</< 




-1 , 0</< 

m 
T. 

-1 0<x</-1 0<y</-1 and p„(x,y) is a point that belongs to rectangle M{ii + x, ji + y). 

[0044] Next, a maximum area rectangle is extracted (Step 604). The grid map obtained in Step 602 is transformed 
into a grid distance map by repeated computations with: 

and 

{<*,(/j)}„ = min({< (tf (/,y-1 )}„.,. {< ltf (/-1. ])}„_, .{< d ('7 + 1W 

<f{M d grid (i.j)} n _, = {</ d ('J)},, = (1 1 } 

[0045] From a seed of the grid distance map, (a,b)= ^ which has the largest distance in the map, 

region growing is carried out in every direction to satisfy tK'eWiSn rfjij) > 0 until the maximum area rectangle is 

extracted 

[0046] Next a texture sample having a desired size is obtained (Step 606). The maximum area rectangle obtained 
in Step 604 is fit to the desired size of the texture sample by wrapping. Alternatively, the maximum area rectangle may 
be fit to the desired size of the texture sample by mirroring. In this case, orientation of the texture should be maintained 

constant after the mirroring. . 
[0047] Referring to Figure 1 , after Step 102, colour and texture distances are weighted with predetermined we.ghting 
factors (Step 104) In the present embodiment of the image retrieval method according to the present invention, colour 
and texture vectors of the data images are compared with those of. a query image in the respective vector spaces to 
separately calculate the colour and texture distances. Referring to Figure 7, the calculated colour and texture distances 
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are projected onto 2-D vector spaces with weighting factors. The feature values on the 2-D vector spaces are projected 
onto a 1-D distance space by considering human perceptual attributes. 

[0048] In particular, the distance between the colour vector of a query image region and the colour vector of a data 
image is defined by the distance function of formula (2) above. A weighting factor applied in this step is determined 

5 according to the colour distribution. For example, if a dominant colour is present, colour reliability is great. On the 
contrary, if a plurality of representative colours are present in similar proportion without a dominant colour, colour 
reliability is low. The colour distribution is expressed by the entropy function. When the colour distribution is uniform 
over an image region, the entropy function has a larger value. When the colour distribution is concentrated on a small 
portion of the image region, the entropy function has a smaller value. Based on this, a weighting factor to the colour 

10 feature is defined as: 



6) 

15 f=l Fqi 



30 



35 



= i-I ^iog I0 (-r) -02) 



where p qi is the percentage of the i-th representative colour in the query image region. The colour-weighting factor « c 



is set to 0.1 if co c < 0.1. 



20 [0049] For the texture feature vectors of the query image region and the data image, texture distance is determined 
using the function having formula (8). in the same manner as for the colour distance. When the method described with 
reference to Figure 6 is applied to take a texture sample, texture reliability may be low if the texture sample is small. 
Although the size of the extracted texture sample can be fit to a desired size by wrapping, unsatisfactory texture features 
may result with increased number of wrappings, due to discontinuity at the wrapping boundaries. Accordingly, in the 
25 present embodiment, each initial texture distance d t {l q J^) is multiplied by a weighting factor. 

d' t (l q Ji)=a 1 d t (l q J,) (13) 



40 



where \ q denotes the query image or query image region, s denotes the texture sample of a desired size, A(-) denotes 
the area of the texture sample of a desired size, count( ) is the number of wrappings done to obtain the desired size 
of texture sample, and a is a constant. 

[0050] In a similar way. a weighting factor to the texture feature is defined as: 

f 1 r ...(14) 

a 



where A( ) denotes the area of the texture sample of a desired size. 

[0051] Next, the distance between the query image region and data image is calculated by combination of the colour 
and texture distances weighted by considering human perception attributes (Step 106). As shown in Figure 7, the 

45 colour and texture distances weighted with the respective weighting factors in Step 104 are projected onto a 1-D 
distance space through a projector that is based on the human visual perception attributes. To combine the two discrete 
features, each weighted colour and texture distance need to normalise into similar scales. The normalisation is per- 
formed using Gaussian normalisation in the present embodiment. Under the assumption that sequence v k is a Gaussian 
sequence, normalisation is performed within the range of [!1, 1] by formula (15) using the mean value u k and deviation 

50 " 0/f of sequence with a 99% confidence factor. Next, the result is normalised within the range [0,1] by formula (16). 
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■ a y m.k + 1 (16) 
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[00521 According to this normalisation method using formula (16). values less than 0 or greater than 1 are set to 0 
or 1 , respectively. This method is effective to prevent the range of the values to be normalised from be.ng narrow due 
to the presence of extremely large or small values. _ 
[0053] For the colour distance function, the distance between colour vectors is meaningless if a data image has no 
similar colour component to the query image, and a deviation in colour distance is small between candidate data 
images On the contrary, the distance between texture vectors has a great deviation between cand.date data images, 
and thus the mean value u, and deviation a, of texture distances is updated by excluding the largest texture d.stance 
until the following condition (1 7) is satisfied, and then the texture distance is normalised using the updated mean value 
ia f and deviation of- 



40 



[QOSA]* The^onSised colour and texture vector distances are combined by the following function (1 8) by weighting 
the colour and texture vector distances with weighting factors: 



where W„ = - — , andlV. = - — 

[00551 Nefle II {fata image sfasilaPlo the query image are determined using the combined colour and texture distance 
Step 108) As can be inferred from the function (18), when one of the texture and colour distances is much smaller 
than the other, each of the texture and colour distances is weighted with different weighting factors, which reflects well 
the human visual perception mechanism in which similarity is perceived by one influential texture or colour feature, n 
the image retrieval method according to the present invention, each multidimensional feature vector is Projected onto 
respective 2-D vector spaces each having distance and weighting factor as elements, and then onto a 1-D distance 
space that defines a combined distance of discrete feature vectors. Accordingly, overall computations are relatively 
simple In addition, the projector used in the image retrieval method according to the present invention, which uses a 
small number of weighting factors which adaptively vary depending on , for example, the distribution of feature vectors, 
enables colour and texture vector based image retrieval according to human visual perception. 
[00561 As described above, according to the present invention, colour and texture features extracted from each data 
image region segmented by colour are separately projected as feature vectors onto respective 2-D vector spaces each 
having elements of weighting factor and correspondingfeature vectordistancefroma query image, and in turn projected 
onto a 1-D distance space based on human visual perception attributes. Thus, the image retrieval according to the 
present invention closely matches the human visual perception mechanism, and overall computations are relatively 

SosiT To verify performance of the image retrieval method according to the present invention, simulation tests were 
carried out using 100 image samples (384H256 or 256H384) designed by Corel Corporation. After segmenting the 

45 image samples into 2456 image regions in total using a colour based segmentation algorithm, and co our and texture 
vectors were extracted from each segmented region using colour vector and texture vector extraction algorithms. Here, 
colour feature vectors were extracted in the LUV colour space, and texture feature vectors were extracted by applying 
a Gabor filter having 4 frequency channels and 6 orientation channels to texture samples of a 64H64 size. As a result, 
forty-eight texture vector components were extracted from each region. . 

so [00581 Figure 8A shows examples of data image regions for a given query image region, which are retrieved by 
combining colour and texture features. Figure 8B shows examples of data image regions for a given query image 
region which are retrieved based on only colour feature. Figure 8C shows examples of data image regions for a given 
query image region, which are retrieved by considering texture feature on the candidate data image regions having 
significant colour distances. Referring to FIGS. 8A through 8C, data images similar to the respective query images 

55 (here fish's bodies) are retrieved based on data image regions, i.e., fish's body regions, in wh,ch both colour and 
texture of the scales are distinct. The resultant data images are satisfactory in view of human perception attributes. 
Thus, according to the present invention, high-level features of an image can be effectively expressed by the appropriate 
combination of low-level features, colour and texture. 
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[0059] In the image retrieval method according to the present invention described above, data images similar to a 
query image can be retrieved based on the human visual perception mechanism by combining both colour and texture 
features extracted from image regions. In particular, the image region based retrieval enables accurate retrieval of 
many objects and many kinds of information from a single image. The distribution of lower-level features, colour and 
5 texture, in the 2-D vector spaces is projected onto a 1-D distance space, and thus higher-level features can be effectively 
expressed with a small number of weighting factors and reduced vector computations. 

[0060] An apparatus for performing a method according to the present invention comprises a computer system im- 
plementing a database containing a plurality of images. The computer system includes means for receiving a query 
image and is programmed to calculate the feature distance between the query image and at least some of the images 
10 in the database as described above. If the feature distance meets a predetermined criteria, the corresponding image 
in the database is retrieved as a search result. 

[0061] The image retrieval method according to the present invention may be embodied as a program stored on a 
computer usable medium, including but not limited to storage media such as magnetic storage media and optically 
readable media. The program codes may be transmitted by electrical or electromagnetic signals. A functional program, 
15 code and code segments, used to implement the present invention can be derived by a skilled computer programmer 
from the description of the invention contained herein. 
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Claims 



A method for retrieving a data image similar to a query image using a feature distance calculated by combining 
color and texture distances by considering human's visual perception attributes. 

2. A method of searching a database of images, the method comprising calculating a feature distance between a 
25 query image and an image in the database to obtain a measure of the relevance of said image in the database, 

said feature distance is calculated by combining colour and texture feature distances, characterised in that the 
colour distance and the texture distance are weighted by respective functions of both the colour entropy of the 
query image and texture sample areas such that the relative contributions of the colour distance and the texture 
distance to said feature distance substantially correspond to the relative importance of colour and texture in human 
30 perception of the similarity between said images. 

3. A method for retrieving a data image similar to a query image in an image database containing a plurality of data 
images, the method comprising: 

35 (a) calculating color and texture distances between a query image and each data image in the image database; 

(b) weighting the calculated color and texture distances with predetermined weighting factors; . 

(c) calculating a feature distance between the query image and each data image by combining the weighted 
color and texture distances by considering human's visual perception attributes; and 

(d) determining the data image similar to the query image using the feature distance. 



4. The method of claim 3, before step (a), further comprising: 

(pa-1 ) segmenting the query image and each data image into a plurality of regions using color features; and 
(pa-2) determining sample regions in the query image and each data image for extraction of texture features. 

5. The method of claim 4, wherein step (a) comprises: 



(a1) generating color vectors of the plurality of regions using the color features and calculating color distances; 
and 

so (a2) generating texture vectors of the plurality of regions using the texture features and calculating texture 

distances. 

6. The method of claim 4, wherein step (pa-1 ) comprises quantizing color vectors of the query image and each data 
image. 



The method of claim 6, wherein quantizing the color vectors comprises: 

(pa-1-1) performing peer group filtering on the query image and each data image for noise removal and smooth- 
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ing effects; and iL _ , , , . 

(pa-1-2) clustering filtered pixel values of the query image and each data image using the generalized Lloyd 

algorithm. 

5 8. The method of claim 6, further comprising: 

defining a J-value indicating color uniformity in each pixel of the query image and each data image which have 
undergone quantization; 

storing the J-value in each pixel of the query image and each data image to obtain J-images; 
io segmenting each J-image into a plurality of regions by a predetermined segmentation method, 

repeating the segmentation of each J-image to obtain a map of over-segmented regions for each J-image; and 
obtaining a final map for each J-image by merging the over-segmented regions based on correlation of color; 

9. The method of claim 8. further comprising indexing a feature vector space by a representative color and its per- 
15 centage in each segmented region. 

10. The method of claim 7. wherein step (pa-1-2) comprises applying a predetermined algorithm to increase the number 
of resulting clusters or to merge the resulting clusters. 

20 11 The method of claim 5. wherein the color features are expressed by a color feature descriptor f c (l k ) with a repre- 
sentative color vector and the percentage of the representative color vector for each region. 

1 2. The method of claim 1 1 , wherein the color feature descriptor f c (l k ) is expressed by: 
25 f c (l k ) ='{(c ((1 .P w ).(e*2.P*2> (cWk-Pmfl 

where k is a positive integer indicating the serial number of each region, l M is the i-th representative color vector 
of the k-th region (/ = 1 .2....N). P U is the percentage of the i-th color representative color vector ,n the k-th region, 
30 and N k is the number of the representative color vectors in the k-th region. 

13. The method of claim 9, wherein indexing the feature vector space comprises: 

assigning representative colors to grid points in a color space having a grid structure; and 
35 storing the result of the assignment as a table in a database. 

14. The method of claim 12, wherein the color distance between each color vector of each region of the query image 
and each color vector of each region of each data image is calculated using: 



40 



N, » 2 *2 

dcUxM = Z Pu + Z pI - Z Z 2a »i} P"P*j 

1=1 (=1 1=1 7=1 



where supposing that T d is the maximum distance by which similarity of two colors is determined, a is a prede- 
termined scalinsbcoefficient. d max = a T d , and d, is the Euclidean distance \\c, - cj| between two color vectors c, 

and cj, a (J = 1 - -r^- * d,y ^ T d- and a /y = 0 if d <j > T d- 
max 

50 15. The method of claim 5, wherein step (a2) uses the Gabor function. 

16 The method of claim 15, wherein, in step (a2), the texture vectors of the plurality of regions are generated using 
' the Gabor function having N frequency channels and M orientation channels, where N and M are predetermined 
positive integers. 

17. The method of claim 16. wherein the texture features are expressed by a texture feature descriptor^/*): 
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where m^- is the means value of the pixel values of the i-th frequency channel and j-th orientation channel for 
region l k , and o kij is the deviation of the pixel values of the i-th frequency channel and the j-th orientation channel 
for region i k . 

18. The method of claim 17, wherein, in step (a2), the texture distance between each color vector of each region of 
the query image and each color vector of each region of each data image is calculated using: 



10 
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jV m 



N M 

+ EZ 



°Uj ~ °2ij 



*<P 9 ) 



19. The method of claim 4, wherein step (pa-2) comprises: 

20 (pb-1) obtaining a grid map of the query image and each data image; and 

(pb-2) obtaining a texture sample of a desired size for each region based on the grid map. 

20. The method of claim 19, wherein step (pb-1) comprises: 

25 (pb-1-1) taking a rectangle M(iJ) having largest side lengths for a sample region, where 0< i< n and 0<j<m: 

(pb-1-2) dividing the rectangle M(ij) into sub-rectangles each having a / x / size; and 

(pb-1-3) obtaining a grid map for the sample region using a predetermined function which outputs 1 if all the 
divided grids belong to the sample region, and otherwise, outputs 0, the predetermined function expressed by: 
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jl if Py(x*y) C I k forV(x 9 y) 
M~*<fJ>\ Q otherwise 



where 0 < / < 
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- 1, 0 <j < 



[f 



-1,0<x</-1,0<y</-1,and p (i {x,y) is a point that belongs to rectangle M(ii + x, jl + y). 

21. The method of claim 20, after step (pb-1-3), further comprising: 

(pb-1-4) transforming the grid map obtained in step (pb-1-3) to a grid distance map by repeating computations 
with: 
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{M d gfid (ij)} 0 = M grid (ij), 



and 



if {< (d ('.y)} n -i = {^ria('V)}„. A^rta('j) = {<,«('•>)}*. 



and 



(pb-1-5) performing region growing in every direction from a seed of the grid map. which has the largest 
distance in the grid and is expressed by (a.b) = ^^J^ to satisfy the relation I^Ji.j) > 0 until the max- 

imum area rectangle is extracted, 

22. The method of claim 21, further comprising fitting the maximum area rectangle extracted in step (pb-1-5) to the 
desired size of the texture sample by wrapping. 

23. The method of claim 21, further comprising fitting the maximum area rectangle extracted in step (pb-1-5) to the 
desired size of the texture sample by mirroring. 

24. The method of claim 4, wherein step (b) comprises: 

(b-1) placing the color and texture distances in 2-dimentional vector spaces defined by the respective color 
and texture distances and their predetermined weighting factors; and 

(b-2) projecting the result of the projection onto the 2-dimensional vector spaces onto a 1 -dimensional distance 
space based on the human visual perception mechanism. 

25. The method of claim 3, wherein the predetermined weighting factor to the color distance is determined based on 
the distribution of representative colors. 

26. The method of claim 24, wherein the predetermined weighting factor to the color distance is determined by: 

N 1 

<y. = i-TiP* i Io Sio(— ) 



C *—i * m r? 



where p qi is the percentage of the i-th representative color of the query image region. 
27. The method of claim 24, wherein the texture distance is defined by: 

(^l^countd^-D 

c/' f (/ Q ./ 1 ) = a 1 d t (l q ,l,) 

where L denotes the query image or each region of the query image, s denotes the sample region, A( ) denotes 
the area of the sample region of a desired size, count( ) is the number of wrappings done to obtain the desired 
size of sample region, and a is a constant. 

28 The method of claim 24, wherein the predetermined weighting factor to the texture distance is determined based 
on the area of an initial sample region extracted from the query image and the area of an sample region extracted 
from each data image. 

29. The method of claim 28, wherein the predetermined weighting factor to the texture distance is determined by: 
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to, = 



10 



15 



20 



where l q denotes the query image or each region of the query image, s denotes the sample region, A(-) denotes 
the area of the sample region of a desired size, count{ ) is the number of wrappings done to obtain the desired 
size of sample region, and a is a constant. 

30. The method of claim 24, before step (b-2), further comprising normalizing each of the color and texture distances. 

31. The method of claim 30, wherein normalizing each of the color and texture distances is performed using Gaussian 
normalization. 

32. The method of claim 31 , wherein normalizing each of the color and texture distances comprises: 
under the assumption that yiat sequence \> k is a Gaussian sequence, performing normalization within the 



range of 1] using \> mk = 



rn.k ' ■ k 



based on the means value \s k and deviation o k of sequence ; and 



l - j ^ in.n ... v 

mapping the result of the normalization into the range of [0, 1] using v mM = - 



25 



33. The method of claim 32. wherein normalizing each of the color and texture distance comprises normalizing the 
texture distance by updating the nneanjS value u r and deviation o t of texture distances by excluding a largest texture 
distance until the condition of k x— <-L, where k is a constant, is satisfied. 

34. The method of claim 24, wherein step (b-2) comprises projecting the color and texture distances onto a 1-dimen- 
sional distance space using: 



30 



d(f 9 ,I t )= W e d c V 9 ,T t ) 



f, 2 , _, <*;(/„/,) ) 

1+ — tan 



{ 2 

y „ tm rf;(/,,/,)J 



where W c = _ and W t = — 
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FIG. 4 
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FIG. 6 
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